Collecting a Ground Truth Dataset for OpenStreetMap

The quality of OpenStreetMap (OSM) and volunteered geographic information (VGI) in general has already been discussed extensively in the literature. Researchers have looked at this issue from different angles such as credibility [2], trust [1], provenance [12, 9], precision [4], and communities [5]. Comparative studies often use commercial datasets or datasets from a national mapping agencies for reference [4, 3, 6]. However, in order to fully evaluate how well OpenStreetMap reflects the streets, buildings, and different kind of amenities out there, such reference datasets are not sufficient, as none of them has a scope as broad as OSM. In a recent study, we therefore decided to collect a ground truth dataset by hand [7]. In this case, the goal was to evaluate whether it is possible to assess feature quality based on provenance information; however, any other kind of study on the quality of OSM data (and VGI in general) is facing the same problem: How to obtain reliable reference data that have the same thematic and spatial scope as the VGI dataset under consideration? In the remainder of this abstract, I will therefore discuss different options to collect such a ground truth dataset.